Logical and Physical Data Independence for Native Scientific Data Repositories

Many datasets in the physical sciences, especially the results of simulations, are defined over a topological grid structure. Applications in these domains would benefit from a principled interface to gridded datasets via a specialized data model. Traditionally, benefits of a data model are realized only after data is ensconced within a managed database environment. However, massive bulk-loading and reloading operations in large-scale data repositories are prohibitively expensive. Instead, we superimpose a specialized data model over native data repositories stored on directly on OS filesystems rather than managed by a database system. Views in a specialized data model can be defined via references to native directory structures and file content, providing physical and logical data independence. This non-intrusive approach appears to reduce space requirements, speed development, and cooperate with legacy applications.